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ABSTRACT: This paper investigates how first-year engineering undergraduates and their 
instructors describe the potential for learning analytics approaches to contribute to student 
success. Results of qualitative data collection in a first-year engineering course indicated that 
both students and instructors emphasized a preference for learning analytics systems to focus on 
aggregate as opposed to individual data. Another consistent theme across students and 
instructors was an interest in bringing data related to time (e.g., how time is spent outside of 
class) into learning analytics products. Students' and instructors' viewpoints diverged regarding 
the "level" at which they would find a learning analytics dashboard useful. Instructors remained 
focused on a specific class, but students drove the conversation to a much broader scope at the 
major or university level but in a discipline-specific manner. Such practices that select relevant 
data and develop models with learners and teachers instead of for learners and teachers should 
better inform development of and, ultimately, sustainable use of learning analytics-based models 
and dashboards. 
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1 INTRODUCTION 

Expanding access, reducing costs, and enhancing quality are among higher education's greatest 
challenges. To address those challenges, the U.S. Department of Education (2015), for example, 
emphasizes the need for the entire educational system to become more "data-driven." As technological 
tools and resources on campuses have grown exponentially, colleges and universities have access to 
more data than ever before, and leveraging such diverse, existing data can provide new information 
from which institutions can learn to enhance the educational conditions related to student success (Baer 
& Campbell, 2012). Such "academic analytics" approaches have focused on the "intersection of 
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technology, information, management culture, and the application of information to manage the 
academic enterprise" (Goldstein, 2005, p. 2). Using technology, the approach brings together large data 
sets, statistics, and modelling to better inform decision-making. Although administrative decisions such 
as enrollment management have prevailed to date, the more specific learning analytics also improves 
teaching, learning, and student success (Campbell, DeBlois, & Oblinger, 2007). 

Indeed, both vendors and universities are trying to understand the long-term learning and decision¬ 
making effects of tools such as dashboards. Although frameworks for learning analytics specifically 
include a variety of stakeholders (e.g., Greller & Drachsler, 2012; Chatti, Dyckhof, Schroeder, & Thues, 
2013), the voices of students and teachers — ultimately the beneficiaries and users of learning analytics 
models or dashboards — are sometimes overlooked throughout the development process (Corrin & de 
Barba, 2014, 2015; McPherson, Tong, Fatt, & Liu, 2016; Newland, Martin, & Ringan, 2015). How students 
may want to interact with data or how faculty can use data to shape their teaching are important 
considerations. Practices that select relevant data and develop models and dashboards with learners 
and teachers instead of for learners and teachers may better inform development of and, ultimately, 
sustainable use of learning analytics products. Moreover, it is well-known that disciplinary contexts 
organize activities, undergraduate curricular content and goals, and teaching and learning practices, 
among many other experiences (e.g., Hora & Ferrare, 2013; Neumann, Parry, & Becher, 2002; Nelson- 
Laird, Shoup, Kuh, & Schwarz, 2007; Smart & Ethington, 1995; Stark et al., 1990). Yet, traditionally 
learning analytics research has focused on the broader university level as opposed to understanding 
specific disciplinary contexts (McPherson et al., 2016). 

This paper contributes to the learning analytics field in both of these areas. We follow a user-centred, 
qualitative research approach by interviewing students and instructors who all engage in a common 
first-year engineering program. Our study extends the learning analytics literature to this specific 
context and seeks to shed insight on the following overarching question: How do first-year engineering 
undergraduates and their instructors describe the potential for learning analytics approaches to 
contribute to student success? Our findings from these end users can help inform decisions for learning 
analytics designers and developers from the outset, which will likely enhance students' and instructors' 
later interest and engagement with learning analytics systems, in particular as the learning analytics field 
moves from a broader, "one size fits all" approach to one that better accounts for disciplinary nuances. 

2 LITERATURE REVIEW 

Learning analytics provides a means for leveraging large, existing data sets to understand educational 
systems and has been used to assist decision-making (Bientkowski, Feng, & Means, 2012), improve 
teaching, learning, and student success (Campbell et al., 2007), and reduce the cost of delivering 
educational experiences (Greller & Drachsler, 2012). Malcom Brown, Director of the EDUCAUSE Learning 
Initiative, noted "the use of 'big data' affords much more nuanced and timely insights into all kinds of 
learning processes" (Brown, 2015, p. 18). Not only does learning analytics enable applied research, it has 
a pure research function leading to understanding how people learn and develop (Baepler & Murdoch, 
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2010; Romero & Ventura, 2010) and how a variety of experiences relate to student success (Bientkowski 
et al., 2012). 

Existing learning analytics systems vary greatly. Purdue University's Signals project applies analytics to 
create course-level student success algorithms, send intervention messages, and develop strategies to 
identify at-risk students. Evaluations of the tool are mixed, but administrators and faculty have 
highlighted benefits for learning and supporting student success (Arnold, 2010). In other examples of 
STEM-specific applications of learning analytics, the focus field of this study, there are learning analytics- 
based early warning systems for an undergraduate mentoring program (Krumm, Waddington, Lonn, & 
Teasley, 2012), and goal-oriented visualizations for a problem solving and design course (Santos, 
Govaerts, Verbert, & Duval, 2012). A primary goal of such systems is to enhance learning by supporting 
learners through the creation of visualization tools, most often referred to as learning dashboards, with 
the explicit intent of improving learning, instruction, and decision-making (Duval et al., 2012). 
Researchers and developers of such dashboard tools cite increased student motivation, autonomy, 
effectiveness, and efficiency of learners and teachers as important drivers (Buckingham Shum, Gasevic, 
& Ferguson, 2012). Learners can observe their performance, compare their performance to a standard 
or goal, and then react and respond to the perceived differences through the creation of a feedback 
loop (Schunk & Zimmerman, 1998). 

What this potential means broadly and practically to students and instructors in terms of learning and 
the learning experience is still up for debate. As George Siemens suggested, "The most significant 
challenges facing analytics in education are not technical. Concerns about data quality, sufficient scope 
of the data captured to reflect accurately the learning experience, privacy, and ethics of analytics are 
among the most significant concerns" (2013, p. 394). Indeed, a recent special issue of the Journal of 
Learning Analytics focused on ethics and privacy in learning analytics. To enhance understanding around 
some of these issues, Madhavan and Richey (2016) assert that user decision-making should be at the 
centre of the design process for big data analytics use in learning. Other authors in the learning analytics 
field have similarly identified the need for understanding potential users' perspectives throughout 
learning analytics development (e.g., Corrin & de Barba, 2014, 2015; McPherson et al., 2016; Newland et 
al., 2015). 

Feng, Krumm, Bowers, & Podkul (2016) recently expanded upon this view that practitioners' voices tend 
to be missing from learning analytics research and argued for more researcher-practitioner 
partnerships. The authors claimed that practitioners' needs should drive the collaborative inquiry, and 
users should be engaged early on in the process. Such early engagement allows researchers to gain a 
strong understanding of school contexts, which is paramount for a learning analytics tool to be effective. 
More importantly, engaging practitioners early ensures that learning analytics researchers address 
meaningful questions that can provide results that may be acted upon by teachers. 

Some recent examples of research have incorporated instructor perspectives. AN, Asadi, Gasevic, 
Jovanovic, and Hatala (2013) worked with teachers in an online educational environment. The 
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researchers sought to determine which set of tools worked well for instructors, teaching assistants, and 
research analysts to understand how educators perceived the ease of use and usefulness of a learning 
analytics tool for their online courses. As a result of involving key stakeholders from the outset in 
developing the system, there was more widespread adoption by instructors. Similarly, Bakharia et al. 
(2016) worked directly with teachers in three Australian institutions to understand how learning 
analytics might assist teaching in online and blended learning environments. Using a combination of 
interviews and user scenarios, the authors developed a conceptual framework in which the teacher 
plays a central role in providing contextual knowledge on the potential usefulness of learning analytics 
tools as well as in the interpretation of user results. Learning analytics research focused on traditional 
face-to-face learning environments has also pointed to the importance of continued engagement with 
instructors. For example, McKay, Miller, and Tritz (2012) provide "coaching" messages to students 
enrolled in an introductory physics course and asserted that relying solely on available data stops short 
of completing the data-expertise-feedback loop. Involving faculty members — or other relevant 
stakeholders — throughout development is essential for completing this loop and reaching ultimate 
success. Thus, across multiple kinds of learning environments, the literature demonstrates the key role 
that teachers should play throughout the learning analytics research and development process. 

In addition to drawing on teachers' views, student perspectives should also be incorporated since they 
too are key users of learning analytics products. In a recent study, McPherson et al. (2016) conducted 
student focus groups across a range of disciplines at an Australian university to understand the provision 
or use of data. The authors considered student perspectives on data use with respect to curriculum, 
pedagogy, and assessment for both epistemic relations and social relations and found that students 
have a range of perspectives on the kinds of data that would be meaningful. Recognizing that the 
learning analytics field tends to take an interdisciplinary, "one size fits all" approach, the researchers 
also intentionally sampled students in different fields so that they could understand disciplinary nuances 
and called on other researchers to adopt such recognition in study designs. 

Our research applies similar overarching methods within a traditional, face-to-face engineering 
educational environment, which characterizes most large engineering institutions in the United States. 
Much like the McPherson et al. (2016) study, we sought to understand student perspectives on the 
provision or use of data and, aligned with the authors' assertion, focused on a single disciplinary context 
as opposed to taking a broader sampling approach. And like the previously described investigations, we 
also sought to understand the perspectives of instructors involved in the same engineering program. 
Thus, our research joins and builds on existing learning analytics literature by 1) seeking to understand 
both student and instructor perspectives on data use and learning analytics approaches, 2) within a 
focused disciplinary environment, 3) with "data" defined much more broadly than only online learning 
management system data, which has been the focus of many prior studies. 
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3 DATA AND METHODS 

This study took place at a Mid-Atlantic university in the United States that enrolled approximately 1,500 
first-year, general engineering students at the time of data collection. In this paper, we report on the 
results of data collection and analyses from two different kinds of participants: students and instructors 
who were stakeholders within the same first-year engineering course. The compulsory, two-course 
sequence sought to facilitate a design-based, active learning environment in which students worked on 
projects within teams. Structurally, the course was divided into different sections of approximately 30 
students and met two times each week for 75 minutes apiece. The 20 instructors who taught within the 
class followed a common curriculum throughout the two-course sequence. 

3.1 Student Data Collection 

We conducted a three-hour session with a group of eight students enrolled in this first-year engineering 
course. The first two hours consisted of a semi-structured focus-group-style interview session to gather 
information on participant perceptions of student data use and learning analytics. During the last third 
of the session, students split into four teams and produced ideas for design of a learning dashboard 
during which they were given opportunities to express and discuss their insights on data visualizations, 
data streams and variables, and how these might merge together to create a dashboard to satisfy 
different student users. This paper only reports on data generated during the first two hours of the 
session, but participants had advance notice via IRB-approved recruitment materials and during initial 
introductions that the design part of the session was forthcoming; we do not believe that this advanced 
knowledge influenced the focus group conversation. 

Researchers in attendance helped lead portions of the discussion, took field notes, and observed the 
design session; team members included: one faculty member in engineering education, one faculty 
member from visual arts, one Master's student from computer science, one doctoral student from 
instructional design and technology, and four doctoral students from engineering education. The 
session's protocol gathered a broad spectrum of information related to student perspectives on data 
and their engagement with educational technology, which could provide insight on how existing data 
might map onto student activity. In developing the protocol, we sought to cover the dimensions of 
Greller and Draschsler's (2012) learning analytics framework, which organizes factors that learning 
analytics designers should consider as they complete their work responsibly, inclusively, and in an 
educationally sound manner. These dimensions include 1) stakeholders (i.e., data subjects or data 
clients, which in this case refers to student interview participants), 2) objective (i.e., reflection versus 
prediction goals of learning analytics approaches), 3) data (i.e., including open and protected indicators), 
4) instruments (i.e., technology, data visualization, and pedagogic theory), 5) external limitations (i.e., 
privacy and ethics), and 6) internal limitations (i.e., data users' required competencies for interpretation 
of data). The protocol used to guide the interviews is shown in Figure 1 and displays how different 
discussion topics map onto those dimensions. 
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Figure 1. Protocol for semi-structured focus group with students 1 


Success [objective] 

1. How do you define success in college? 

2. What are some important keys for success as a student? 

3. How might you go about measuring success? 

4. How might you go about measuring your experiences at [INSTITUTION]? 

5. How much interest do you have in knowing how well you are doing as a student? 

a. Individually within a specific course? b. Holistically within the institutional environment? 
Data/Information [data; instruments] 

1. What kinds of data do you think [INSTITUTION] has on you? 

2. What types of data do you think would help you understand how well you are doing as a student? 

a. Individually within a specific course? b. Holistically within the institutional environment? 

3. What information would you like to have known about transitioning from high school to college? 
Comparisons/Influences [objective; internal limitations] 

1. In what ways would knowing your performance relative to your peers be helpful? 

2. How do your peers influence those decisions? 

3. How do you make decisions about how to study or spend your time? 

4. Who or what influences your time management? Why? 

Credibility/Trust [external limitations] 

1. Where do you stand on individual data privacy concerns? 

2. What information would you feel comfortable about faculty knowing about you? 

3. Who would you be ok with being able to access information about your education? 

a. Performance information? b. How you spend time? c. How you engage in and out of class? 
d. Parents/families/teachers/peers/group members/RA's/faculty/admin? 

Technology [objective; instruments; data; internal limitations] 

1. How do you use technology in your everyday life? 

2. What do you use the LMS for currently? 

3. What functions of the LMS would you like to see used more broadly? 

4. How could technology help track your academic/non-academic experiences? 

5. What types of persuasive technology (think Nike fitbit, mobile notifications) would help you to 
perform better academically? 

1 Dimensions of Greller and Drachsler's (2012) learning analytics framework shown in brackets 

We analyzed data collected during this session in a multi-step process. All participating researchers met 
to review field notes and identify initial impressions and consistent themes that emerged across the 
team. Data were also transcribed and imported into NVivo, a qualitative data analysis software package. 
Both faculty members, the computer science doctoral student, the instructional design and technology 
doctoral student, and an engineering education doctoral student all took an initial pass through the data 
using an inductive, constant comparative method (Patton, 2002; Robson, 2011). As the nature of this 
study was exploratory and did not seek to test an existing framework, we did not impose a coding 
scheme but instead allowed codes and themes to emerge from the text. The full team came together to 
discuss codes and develop a common codebook. Sections of the transcript were then fully coded using 
the new codebook by both faculty members and the instructional design and technology doctoral 
student — the graduate student coded the entire transcript to maintain at least one consistent coder 
across pairs of coders. Following the individual coding process, both coders met to discuss any 
differences in coded sections of the transcript and settle upon one final coded transcript — because 
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there was a single coded transcript with every individual code agreed upon by coders, there was no 
need to calculate an interrater reliability. Following this process of using multiple coders enhanced the 
reliability and trustworthiness of analysis (Creswell, 2009). 

3.2 Interviews with Instructors 

To collect data from instructors, we conducted individual, semi-structured interviews. Eight of the 20 
instructors in the first-year engineering program replied to an email invitation and consented to 
participate (note: there were no apparent systematic differences in respondents' versus non¬ 
respondents' characteristics). Five participants were full-time instructors, and three were graduate 
teaching assistants who served as instructors of record for their assigned sections of the course. We 
conducted one interview with each participant, ranging in length from 25 to 45 minutes. The interviews 
asked instructors to reflect upon and discuss how data are used or could be used to inform their 
teaching practices, and also inquired about the student data that instructors would like to have to 
inform their teaching practices. The interview protocol was structured similarly to the focus group 
session held with students. Prior to interviews with instructors, the protocol was tested via a pilot 
interview with a graduate student instructor affiliated with the course under investigation; results of 
that pilot interview are not included in this study's findings, and the protocol was adjusted based on that 
session. 

Interviews were audio-recorded, fully transcribed, and coded following a similar inductive process as for 
the student data collection. Two independent, engineering education doctoral students open coded 
three transcripts separately, where the coding level pertained to an interviewee's complete thought. 
The same researchers collaborated to adjust and collapse those codes, which resulted in a common 
codebook. One researcher then coded all transcripts based on the common codebook, and the second 
researcher subsequently performed a peer audit of each coded interview and confirmed, changed, or 
added categories as deemed necessary, which again contributed to the trustworthiness of findings 
(Leydens, Moskal, & Pavelich, 2004). The two researchers discussed any differences in coding until 
commonality was achieved — this iterative process resulted in a single set of fully coded transcripts, 
which, as with the student data, make interrater reliability calculations unnecessary. Following this 
coding process, the two researchers again came together to identify themes that emerged across the 
interviews, and definitions for each broader theme were created. Finally, peer debriefing was employed 
using a third researcher to continue building validity of the resulting themes (Creswell, 2009). 

4 RESULTS AND DISCUSSION 

4.1 Findings from Students 

We arrange the results of the student data collection under four themes, all of which were prevalent 
throughout the focus group session: 1) Students have strong views on who should access their data, 2) 
The importance of time, 3) Students want help making the transition to university life, and 4) Learning 
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analytics dashboards should be discipline-specific. We present supporting qualitative data for each of 
these themes in the sections that follow with summary points for each theme in Table 1 in Section 4.3. 

4.1.1 Students Have Strong Views on Who Should Access their Data 

One portion of the session focused on data privacy and ethical considerations associated with learning 
analytics. Students laid out data that faculty members in particular should not be able to access under 
any circumstance, such as social security number, financial information, and health information. They 
also cited the importance of students maintaining the authority to determine what individual data are 
shared with different institutional stakeholders. As one student noted: "There are some students that 
don't want to share everything with others ... A lot of information is not unique to the learning 
situation. It is a student's responsibility, not a faculty one." Rather, students felt more comfortable being 
able to opt in or out of each aspect of any kind of learning dashboard put into practice: 

[Student 1]: I would opt out of all but, well, other students may feel differently so to provide that 
would be good. [Student 2]: It allows for the data to exist and without forcing all of the students, 
you wouldn't want to but I wouldn't care, so why not? 

Students also described different circumstances where faculty members should and should not be able 
to access academic-specific data. When asked about faculty accessing students' grades in other classes, 
one student noted, "Should they need to have access to that? They shouldn't need to, but I do not think 
it matters if they do." Another student was a bit more cautious about granting access to information: 

So that ultimately comes down to the goal of the faculty member. I do not know who makes the 
call for that. It is bad for a faculty member [to have access to other course grades] since you do 
not want them to treat you differently or stereotype you based on a previous or other work — 
that's wrong. However, they are still an educator so it is their responsibility to know how you are 
doing in your educational experiences as a whole. So in that sense it is critical for them to have 
that information available to them. 

Ultimately, the group of students reached a consensus that faculty members should have access to 
academic data if they were serving in an advising role, but that would not be the case if the faculty 
member only served in a teaching role. 

Finally, we heard varying perspectives from the students on data access based upon the purpose of 
using the data. As one student argued: 

I respect the different views on faculty, but I would almost take it from a different perspective. 

How are they going to be using it? Let's say you guys are doing research and you are doing this 
scenario where you are looking at everything as a whole . . . one of your dots in a million dots; 
that does not bother me. But if it is a teacher that I have, and they are just going to look to see if 
she is bad in all her classes or is she just doing well in this one, or whatever they wanted to know 
... so aggregate versus individual. 
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The privacy concerns highlighted by the students demonstrate the importance of considering learners as 
a key stakeholder in the learning analytics process, consistent with existing frameworks of Greller and 
Drachsler (2012) and Chatti et al. (2013). Although a plethora of data from students may be available, 
use of those data may be inconsistent with students' data access concerns, which may prevent full 
engagement with an end product. 

4.1.2 The Importance of Time 

Students repeatedly pointed to time elements as being an important characteristic of a learning 
analytics product and saw value in incorporating scheduling or time management functions to help 
support their overall success. A scheduling function within a dashboard could be useful in helping 
students maintain sight of the bigger picture instead of focusing on individual courses. As one student 
noted: 


You might have three tests in three different classes on the same day and then two days later 
you have another test. So you are thinking about how you are doing on each individual piece. 

You're not thinking about the big picture. 

Other students expanded upon this idea and described how mobile notifications would be helpful for 
keeping track of due dates and appointments. They noted the utility in having calendar integration 
capabilities where dates and times sent in emails or in class materials could automatically populate onto 
a calendar system 

Beyond basic scheduling, students wanted to know how their personal time on tasks related to their 
classmates' time spent on tasks, historical data, and performance data. Such functionalities that bring 
awareness to those relationships could enhance students' metacognition and their ability to self- 
regulate their learning processes. For example, one student described that knowing how long it took 
students enrolled in previous courses to complete a given assignment provided insight on his own 
understanding of the class material. Students agreed that a dashboard that made such data available 
would be valuable, although there was some disagreement in how such information might be useful. 
One student saw that as an opportunity to identify the appropriate peers to seek out for help: 

If my classmate is spending a half an hour [on an assignment], and I'm spending like an hour or 
more, that's me wasting time . . . that's like one of my biggest pet peeves is wasting time. So if I 
see that it takes somebody thirty minutes, then yeah, I would group up with them, talk to them 
about why does it take me so much longer. . . like you know seeing their perspective on things. 

Contrastingly, another student felt like knowing averages for the class would be helpful as a data point 
but argued that such information was only so meaningful: 

The only thing that I would be interested in for that perspective is knowing just kind of the 
average time on a whole of the class takes to work on it ... I think anybody else's time on it is 
going to be a poor measure of how much progress I need to make on the assignment, and how 
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much time I need to invest in it... I don't think anyone can tell me how much I need to work or 
not work on it on a specific assignment to achieve something. 

As these first-year students were still early in their programs and adjusting to life on their own, they also 
pointed to how such a dashboard could help with that transition, perhaps more so than with their in- 
class learning. The participants were most excited about the potential for a dashboard to coalesce 
information that they could use to make informed decisions about how to spend their time and not 
waste time waiting. For example, one student noted: 

This is going to sound off topic, but going with the tracking, I would love to know how many 
people have recently swiped into [Gym] so I know if it's too busy . . . Also if the washer and 
dryers are full, that would be good [to know]. 

Although such elements were outside the scope of the original intent of our learning analytics 
discussion, the focus group suggested that such elements may be useful for initially drawing students to 
a learning analytics platform. Driving student traffic to the system should be an important consideration 
in its potential for success. 

The importance of time theme also emerged in McPherson et al.'s (2016) study across disciplines within 
the Australian context. Students pointed to time management and scheduling, time spent on resources, 
and time spent by high-achieving students on studying as data that would provide meaningful 
information. The theme of time also is consistent with research regarding student transitions to college. 
Kelly, Kendrick, Newgent, and Lucas (2007) recommend including time management among other skills 
in secondary education transitions programs. These results suggest that student success models may 
need to incorporate a more holistic set of data beyond the traditional course-specific data generated by 
students and captured by institutions. Additionally, such commonality across studies and contexts builds 
evidence for the credibility of our findings as well as for the potential transferability of the importance of 
time to other settings. 

4.1.3 Students Want Help Making the Transition to University Life 

In addition to helping with time management, students pointed to the potential usefulness of a learning 
analytics product in easing the transition into university life, which was also outside the scope of what 
we were anticipating before the session. Students pointed to a variety of "life" ideas that would ease the 
transition from attending high school and living at home to being on campus. A single dashboard 
product would have been helpful during the summer prior to matriculation, as described by one 
student: 


Like a checklist, I could never find one. You know it was a little overwhelming because I didn't 
know. I didn't want to miss something. There are just so many things you have to turn in, your 
health stuff, the financial [forms], contacts, and I just didn't want to miss something and later on 
miss the due date ... I was jumping around going to every little thing to see if it had a due date. 
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Demonstrating the added value of a dashboard to a student before they arrive on campus in this 
manner could potentially be a way to encourage later usage for functions related to learning. 

Once on campus, students pointed to a number of different applications of a dashboard that would help 
ease their transitions to university life. For example, "You're obviously not going to know where any of 
the buildings are . . . just [knowing] the place itself, so like where the dining halls are, or where your 
classes are and all that." Other students pointed to early troubles managing their meal plans because 
they did not have the information in an easy to access location like a dashboard: 

My only real issue was like managing your meal plan. I kind of made a little mistake... like not 
eating steak every day. I went to the little smoothie shop in the [dining hall], Ooh, pricey. So I 
was like oh, I will spend 15 flex dollars a day. [Sigh] It was not great. I was supposed to spend like 
15 actual dollars a day, so like 7 and a half [flex] dollars, so I was out of money by like October. 

Developing a system that helped students recognize these "life" transition misconceptions in real-time 
as opposed to later in the semester would have a real value proposition for students. These results were 
again consistent with Kelly et al.'s (2007) recommendations for secondary education transition 
programs. In total, they advocate a focus on general coping skills, time management, and study skills to 
assist students in their transition to college. These results may indicate that to be useful to students, 
especially in the first year, learning dashboards may need to help students navigate the college 
transition. In discussing challenges related to use of big data in learning, Madhavan and Richey (2016) 
argue for the need to have analyses occur in real-time to provide instant information and feedback to 
users. Our data suggest that assisting students in decision-making across the totality of transition 
challenges in real-time may afford students greater ability to focus on learning. 

4.1.4 Learning Analytics Dashboards Should Be Discipline-Specific 

Student participants emphasized the need for any learning analytics dashboard to be designed 
specifically for undergraduate engineers. Participants consistently pointed to differences in learning 
environments as well as differences in perceived expectations of students in terms of academic rigour. 
As one student noted when describing the potential utility of benchmarking himself to students from 
across the university, "I can't say I think I could compare myself to someone in a different major. I think 
that's kind of comparing apples to oranges." Another student agreed with this sentiment: 

I have a lot of friends that are outside of engineering, and not that I like look down on their 
majors and look down on the work that they've done, but you can see how they might be doing 
really well in this, but it can't really be compared to what I'm doing over here [in engineering]. 

And so that's why I might say that their data is different. 

Even when describing available free time and the potential usefulness in a dashboard helping structure 
and manage outside of class time, student participants continued distinguishing between academic 
disciplines: 
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I mean engineering is obviously going to take a lot more effort and a lot more time than any of 
those majors can, so I'm not sure if you can compare how much free time you have and ask [non¬ 
engineers] for how much free time they've got. 

Although we do not necessarily agree with this assertion, it was evident that these engineering students 
strongly felt that learning dashboards aiming to provide benchmarking data should remain discipline- 
specific to be useful. Students may want additional assurance that the learning analytics model is 
helping them navigate their own personal academic pathway. This discipline-specific theme aligns with 
the previous research by McPherson et al. (2016). In that study, students were interested in data 
emphasizing study habits appropriate to a discipline, discipline-specific resources, and discipline-specific 
procedures (e.g., time spent on readings). It appears as if students in both studies had the mindset that 
educational experiences vary by discipline, and thus learning analytics researchers and developers 
should consider disciplinary differences. 

4.2 Findings from Instructors 

The aim of this part of our study was to investigate experiences of first-year engineering instructors with 
student data. The first broad topic under investigation related to how instructors use data, and the 
second related to the student data that instructors would find useful for helping their teaching. The 
sections that follow describe each of those themes and provide supporting evidence from the faculty 
interviews. A summary of the themes is shown in Table 2 in Section 4.3, which is followed by a table 
displaying the pervasiveness of each of those themes across participants. 

4.2.1 How Instructors Use Data: Academic Process Improvement 

Participants revealed that they use or could use student data to initiate course-level adjustments or 
identify problems within the course. For example, if data could be used to know where and why 
students were struggling, instructors felt they could intervene to help determine the causes of the 
issues. Instructors wanted to create an environment in which students thrive, and they believed that if 
they were able to use data to make adjustments, either across the course or with an individual student, 
then students would be more inclined to engage with the class material and be successful in the course. 
One instructor, for example, wanted to be able to use data to recognize earlier when a student was 
struggling and why they were struggling: 

To be able to identify the students who need more help than others will be very useful. You can 
kind of figure it out after a while to find out who is struggling and who isn't, but you don't know 
why, and you can suggest better studying habits or whatever. Or you can understand if it's a 
motivational issue. 

Another instructor echoed this notion of wanting to understand what is going on when she said, "On a 
general level if I have given an assignment that everybody has tanked, there is an 'oh what happened 
here', was it me, was it you, was it the assignment, what did we miss on this kind of thing." Like the 
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other participants, this instructor wanted to be able to use data to implement process improvements 
within the course to minimize the likelihood of similar poor results on future assignments. 

Another instructor paid attention to how many students visited his office hours for help on homework 
to capture whether or not the students understood the concepts from his teaching. He said, "If there are 
a lot of students coming into the office hours and having difficulty, then I need to go back and spend 
extra time in the class to review what we have done." Like others, this instructor is using a different type 
of data — numbers of students at office hours — to understand how well students are doing, and in 
turn, going back and re-teaching concepts when his informal data collection suggested he needed to 
spend more time on topics. 

This theme of data use for academic process improvement was generally consistent with classroom- 
scale data-driven decision-making frameworks developed for K-12 education. The instructors in our 
study described using a combination of data sources to understand what is happening in the classroom 
and how they could better tailor the learning environment to support their students. Both Marsh (2012) 
and Mandinach, Honey, and Light (2006) present frameworks that describe multi-step processes in 
which data are translated into information and knowledge, allowing decisions and action in a continuous 
feedback loop. Although these frameworks were developed for the K-12 context, the instructors' 
descriptions provide indications that the frameworks may hold in first-year college classrooms as well. 

4.2.2 How Instructors Use Data: Personalized Interaction 

Instructors sought ways to connect with students on a personal level and did so by collecting data 
informally so that they could personalize learning experiences. More personalized learning experiences 
enabled opportunities to cater to students' interests as a whole, which helped instructors make material 
seem more relevant to students. For instance, one instructor noted that she created nametags to 
capture personal information so that she can "try to get a good understanding of the dynamics going on 
and try to use their hobbies as examples in class and those types of things." Likewise, another instructor 
used informal data "to get [the students] to get to know each other better" to "have a learning 
community in the classroom." A third instructor similarly used note cards to get 

... an idea about the climate of the class. When I get their majors, I have an idea about what 
people are interested in; for what people are excited about, I try to make those things happen. 

And for the things students are nervous about, I try to go over them the next class and ensure 
that they are not worried about it for this or that reason. 

It was evident across the participants that instructors wanted to use informal and observational data to 
create connections with students at both the personal and academic levels. They took on data collection 
on their own so that they could get a better understanding for why students wanted to become 
engineers so that they could play to students' ambitions as they presented course material. Within the 
learning analytics field, there is an opportunity to develop mechanisms to capture student motivation 
for studying a certain field in a systematic manner so that instructors can use those data to create more 
engaging learning environments for students. 
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Some of this kind of motivational data has been captured through the use of surveys; for instance, Jones 
(2009) created a survey instrument that had been administered in the class under investigation that 
captured student beliefs about how instructors care about their learning and how they care about 
students on a personal level. However, those survey data often are not analyzed immediately and 
provided to the course instructors during the semester; the learning analytics community could 
investigate how to provide such data to instructors in a more real-time way. Students are not the only 
ones who would benefit from real-time feedback — instructors would benefit as well. 

4.2.3 How Instructors Use Data: Instructor Teaching Improvement 

Not only do instructors want access to data to improve the content of the course and create 
connections with students, they want data to improve their own teaching methods. Data would help 
them understand the changes that occur in their courses "because I want to know what is happening in 
my class" to understand the problem points for students. One instructor, in talking about wanting to 
know whether students were comfortable with a topic or not, stated: 

... if I could see their progression, I could see if my students are making progress towards their 
learning goals, and if they are not, maybe I could see what week did no one change their 
comfortableness with a topic. Maybe I would know that week if I did a good enough job on that 
topic. 

Instructors seemed to have a sense of responsibility to their students with respect to their learning 
progression. As another instructor noted: 

... if I had a real true level of understanding within a class constantly that would let me know if I 
am going on and on about something and no one is understanding what I am saying. Then I know 
I would need to regroup and talk about it differently. 

He, like others, wanted to make sure that students comprehended what he was trying to teach and 
thought that if he had data on students' actual understanding levels, then he would be able to know if 
he needed to change his teaching practices. This theme relates to the instructors themselves and the 
ways they teach concepts to students as opposed to focusing primarily on the actual content. Instructors 
felt they could use data to ensure that they were staying on point and helping students learn; the data 
could allow improvement in their own role as instructors. 

Using data in this way is also consistent with the data-driven decision-making frameworks mentioned 
previously, but in a more personal way. Within this theme, the decisions were with regard to their own 
instructional techniques rather than larger classroom or course level decisions, and the processes 
described coincide with how some K-12 teachers use data. Fives, Barnes, Bratkovich, & Dacey (2016) 
present an in-depth analysis of one teacher's sub- and micro-processes within the larger decision¬ 
making frameworks and provides an example of how teachers use the grading process to simultaneously 
transform student outcome data into information and knowledge that informed teaching improvement 
decisions. Similar processes occurred within the first-year engineering educational environment on an 
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informal basis; there appears to be an opportunity to make this process more systematic using learning 
analytics techniques. 

4.2.4 How Instructors Use Data: Protecting Potential 

The largest sense of uneasiness from interview participants while talking about data related to the 
notion of having data at the individual versus aggregate level. Different from the subsequently described 
"Level of the Data" theme, however, this theme related to potential unintended negative consequences 
of having access to additional student data. Most instructors liked the idea of having data, as long as the 
data did not cause any unnecessary harm to students. One instructor stated, "If I knew who the good 
students were and the bad students were . . . somehow it might change my expectations of students, 
and how they responded to that ... so I am not sure how much I want to know about all that." The 
instructor took a cautionary stance when it comes to data and how data might be presented to her 
about her students, which was echoed by multiple participants. In another example, an instructor was 
adamant when she said, "One of the things I actually don't want is data on individual students [on 
incoming characteristics] because there is a lot of risk in putting students in boxes and stereotyping 
them" and "It is really hard to eliminate preconceptions; once you get them, they do not go away. Like 
knowing this is the kid that got a C in high school calc[ulus] ... it is too easy to start to box people in." 

Participants not only wanted to protect themselves with respect to preconceived notions of students 
that data could produce, but they also wanted to protect students from the likelihood of success data. 
As one instructor stated, "I would be cautious in which data I showed students" so as to not push the 
students into a poor state of mind. He thought that if instructors "focus on motivating the students" 
through positive reinforcement, then better outcomes would result rather than showing the students 
data about the relationships between prior cohorts' background levels and student success. If a student 
was in a "failing" category based on historical data with respect to pre-college characteristics, 
communicating such information to students may have serious negative consequences on their 
motivation and confidence levels in the course. 

This desire to protect the potential of students was a consistent theme across participants. As another 
instructor described, "I wouldn't want to see my students in a limited way. I want to see them in their 
full potential." Participants did not want an all-encompassing view of student data in helping their 
students achieve success and thought that some types of information could potentially limit success. 

4.2.5 Data Instructors Would Find Useful: Accessible Data 

The majority of data that these instructors indicated wanting about their students are data presently 
captured at the institution under investigation and most likely at other institutions. Instructors wanted 
to know the background knowledge of their students as well as some other academic information, such 
as students' prior or current course schedules. One instructor described this desire: 

I guess if the sky were the limit ... I would probably like to know a little more about with what 
they were coming in. I would like to know what they do and part of talking with them in class I 
get to learn some of those things. Some of the training that students have today, with 
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international students, and students out of state, and how the high schools vary from each place. 
Nowadays about a quarter of the population already knows CAD before they come in here, and 
the same thing with the programming. 

Another instructor articulated the same sentiment stating, "I would probably like to know what courses 
they have taken previously, such as high school since this is a [first-year] class. Would have some 
understanding of what their math levels are, if we are doing programming, have they done 
programming before." Two other instructors also aligned with the previous two comments stating, "I 
would like to know how many other classes they are taking," and, "I would like to see how they are 
doing in their other classes, and even in the current semester." 

The participants thought such data elements could provide a more comprehensive understanding of 
current skills and course loads of their students, as opposed to following ad hoc, informal data gathering 
on an individual level. As the title of the theme expresses, most of these data are available within the 
institution but have not been provided to these instructors. One immediately achievable 
recommendation stemming from this study that would complete the research-to-practice cycle for this 
class would be to provide instructors with an overview of enrolled students' class schedules. The 
institution's student data management system could easily be leveraged to turn this desire into a reality. 

The layered nature of this theme from the classroom to the institution and across multiple data sources 
coincides with Mandinach et al.'s (2006) data-driven decision-making framework. Their framework 
layers the decision making process at the classroom, building, and district levels, across multiple data 
sources to include a data warehouse at the K-12 level. This consistency in structure with the institution- 
level data of a university also supports the translation of data driven decision-making from the K-12 
context to higher education. 

4.2.6 Data Instructors Would Find Useful: Data not Readily Available 

Participants were enamored with wanting to know how students spent their time outside of class. 
Instructors wanted to know more about where and how students put their efforts on their academic 
work. As an instructor summarized: 

What I want to know is how students are spending their time. So that is a big thing, on what 
homework problems are students spending their time . . . how much time are students spending 
on the video tutorials . . . how much time are you spending on working on the actual program? 

On the project stuff, I want to know how much time and what they are doing on the actual 
projects. 

Not only did she want a general sense of time on task, she also wanted to know how students were 
putting in time and effort on specific components of coursework. Another instructor similarly articulated 
the desire for knowledge of student time by saying, "I want to know if students are working off-campus, 
and how much time they are spending in a job. This will let me know how much time they can spend on 
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the course." Student time falls within the theme of data not readily available because it is not easy for 
instructors to obtain, or for that matter, not collected systematically within institutions. 

Other types of potentially useful data that participants brainstormed included student engagement, 
motivation, and level of understanding. One participant "wasn't sure how engaged their [students] were 
day-to-day" and wanted to be able to know "whether they were getting [course material] or not getting 
it." Another interview participant commented, "I would like to know [students'] comfort levels [with the 
material] and be able to see that from week to week." Another similarly stated, "I wish there was a way 
to capture students' understanding levels without forcing students to always think back and to check a 
box or something. An actual true level of understanding throughout the room would be great." Some of 
these data types might be easier to capture then others, but none of these data are readily available. 

4.2 .7 Data Instructors Would Find Useful: Level of Data 

After creating a level of understanding on data uses and desires, participants were asked to consider 
whether they would want requested data at the individual or aggregate level. This consideration was a 
struggle for most participants, as several wanted enough information to make decisions but also did not 
want to be flooded with too many unnecessary individual data points. One instructor, for example, was 
uncertain about which level of data she thought would be most helpful: 

At the individual level it would be good to know what they were struggling with. It's one of those 
things I think I would have to experience, knowing about motivation and such at the individual 
level. I think it might be too distracting, and I would focus on one or two students who the data 
say are kind of checked out, and not focus on the other 90% ... I don't know, but at first I would 
like data at the aggregate level. 

Another instructor showed uncertainty as well stating, "If the data were at the aggregate level, we 
wouldn't have the details needed to know students. But you can get information from just talking to 
students." He thought there might be value in having the information at the individual level, but thought 
he could just as easily get that information in other ways, such as talking with a student. These 
perspectives summed up how most interview participants felt — they would rather have most types of 
data at the aggregate level, and not at the individual level, for reasons previously discussed. Consistent 
with the "Protecting potential" theme, these results further support a potential aversion for these 
instructors toward the individualized view of student data. 

4.3 Summary of Results for Students and Instructors 

Four themes emerged from the student focus group session, as summarized by Table 1. They had strong 
views on who should access their data, continued to point to the importance of time in their ideas about 
learning analytics approaches, thought a dashboard could help them with the transition to university 
life, and argued that learning analytics dashboards should be discipline-specific. As summarized by Table 
2, instructors discussed the desire to have certain kinds of data at the aggregate level about their 
students, some of which are readily available within the institution but not provided to instructors 
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directly. Participants appeared to be excited about the notion of using data to improve the student 
experience overall by creating an environment conducive to learning. Table 3 displays a frequency map 
across the eight instructors. Themes that were most prevalent across the highest number of 
interviewees included "Academic process improvement" and "Accessible data"; the least pervasive 
themes across interviewees were "Instructor teaching improvement" and "Data not readily available." 
At minimum, at least half of the participants discussed each theme during the interviews. 


Table 1. Summary of themes from student focus group 


Students have strong 
views on who should 

access their data 

• Certain personal data should never be shared with faculty 

• Aggregate data are acceptable, but not individual data 

• Students differentiated between faculty who are advising versus those only teaching 

• Working with students to determine opt-in and opt-out policies is important 

Importance of time 

• A scheduling function would help students see the bigger picture 

• Information on time can help new college students adjust to independence 

• Students wanted to know how personal time tracked onto historical data 

• A time-based function could spur student use of a learning dashboard 

Students want help 
making the transition 
to university life 

• Can help manage pre-matriculation requirements and deadlines 

• Can help manage finances early (e.g., meal plan use) 

• Can offer some kind of mapping or navigation function 

• A real-time "life" management function could spur student use of a learning dashboard 

Dashboards should be 
discipline-specific 

• Learning environments and perceived expectations differ across different fields of study 

• Comparing time and effort across disciplines would be like comparing apples to oranges 

• Students would not find university-wide benchmarking data useful 


Table 2. Summary of themes from instructor interviews 


+■* 

Academic 

process 

improvement 

Definition: Making changes to content and/or presentation of a course based on 
information from or about students 

• Student data identify the need for course-level adjustments 

• Instructors drew on a variety of data to understand students' understanding 

• Instructors' main goal for drawing on data was to help students succeed in the course 

“O 

Personalized 

Definition: Connecting with students as individuals regarding personal interests and/or 

to 

3 

interaction 

regarding personal academic progress 

to 


• Instructors collected data on an informal basis to connect with students on a personal 

+■» 

u 


level 

i- 

+-» 

V) 


• Informal data collection allowed instructors to bring students' interests into examples 

c 


• Finding ways to collect student motivation data systematically would help instructors 

> 

O 


• Providing such data to instructors in real-time would help them tailor classes to 



students 


Instructor 

Definition: Evaluating personal teaching 


teaching 

• Student data also help improve teaching methods by enabling self-reflection 


improvement 

• Instructors owned problems if data suggest the class at large struggled with concepts 
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Protecting 

potential 

Definition: Ensuring data are not used to harm or discourage any individual 

• Instructors were concerned about the harm data could do to students 

• Stereotype threat was a potential negative implication of expanding student data access 

• Predicting negative outcomes via learning analytics could de-motivate students 

Data Instructors Would Find 

Useful 

Accessible data 

Definition: Data readily available to instructors, captured by the university's current 
systems 

• Most desired data are already captured by the institution 

• Having a "data warehouse" from which instructors could access student data is a desire 

Data not readily 
available 

Definition: Data not available or data that would require considerable work to use 

• Instructors wanted data related to how students spend time engaging with the class 

• Capturing students' conceptual understanding in real-time would help instructors 

Level of data 

Definition: Differentiating between aggregate and individual level of data on students 

• Uncertain about the extent to which they preferred individual versus aggregate data 

• Individual data would be required to help struggling students, but data in aggregate 
seemed to be "safer" and more actionable for teaching 


Table 3. Prevalence of each theme across instructor participants. 


Topic 


Theme 


First-Year Instructor Participants 


Academic process improvement 


How instructors use 
data 


Personalized interaction 
Instructor teaching improvement 
Protecting potential 


Accessible data 


Data instructors would 
find useful 


Data not readily available 


Level of data 



1-4 


Frequency of theme 


4.4 Limitations of the Findings 

This study design includes limitations that could influence the potential results or subsequent 
interpretations. First, eight first-year engineering students and eight instructors from one university 
were participants in the study. As such, we do not claim that we can make sweeping generalizations 
based on this sample size. Because we were conducting exploratory research on students' perspectives 
with respect to learning analytics that have no apparent connection with existing variables collected 
across the entire first year program (e.g., such as demographic information or academic performance), 
there is no way of knowing whether these perspectives are representative of the 1,500 students 
enrolled in the program. Future research could develop and administer an informational survey across 
the first-year student population to investigate the pervasiveness of these themes across the broader 
population. As we have shown throughout our results section, however, our findings align with prior 
studies that conducted similar analyses but did so in different contexts and disciplines. Triangulating 
results across qualitative studies may help the learning analytics community gain confidence in the 
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transferability of these results to other contexts, in particular to other institutions with large first-year 
engineering programs. 

Additionally, the researchers involved with this study were familiar with the instructor participants and 
educational context. Although measures were taken to ensure credibility and trustworthiness, such 
familiarity with the instructors may have influenced how the participants engaged within the interview 
and how the researchers may have interpreted data. Lastly, during the instructor interviews, the 
researcher at times had to provide examples to the participants to initiate conversations. For instance, 
when asking about student data uses, instructors' answers would have been fairly limited if the 
interviewer did not offer that grades on assignments, for example, could be considered data that they 
used to understand student performance. The remaining time with each participant once the 
conversation began did not require such coaxing, but without initiating the conversation with an 
example, the reported findings may have included a theme related to the fact that instructors do not 
consider data. Given how the remainder of the interview proceeded and the array of perspectives and 
kinds of data identified by each participant, we believe that making such a conclusion would be 
unwarranted and, rather, that the initial prodding helped the conversation move past the initial 
awkwardness of interview conditions. However, we do acknowledge that it is possible that such initial 
moments could have influenced participants' responses for the remainder of the interview. 

5 CONCLUSION 

In this paper we followed calls to consider student and faculty perspectives on learning analytics 
approaches within a focused disciplinary context; end users' views often are overlooked throughout the 
development process. Our findings demonstrate how following such a user-focused approach can 
produce important insights regarding design features and data considerations. Such an approach that 
selects relevant data and develops models with learners and teachers instead of for learners and 
teachers should better inform development of and, ultimately, sustainable use of learning analytics- 
based models and dashboards. 

A few salient points were common across student and instructor participants. Both groups emphasized 
the need to protect individuals with respect to student data. Students did not mind being one point in a 
larger cloud of data but were apprehensive about providing their teachers with unnecessary data points. 
Instructors similarly preferred data about their classes in aggregate so that they would not inadvertently 
form stereotypes about individual students. Moreover, instructors were cognizant of the fact that 
providing too much predictive modelling data to students might negatively influence motivation levels. 
If a learning analytics system communicated to students that they would likely not be successful, 
instructors worried about what that might mean for students' subsequent effort. Therefore, our findings 
suggest that learning analytics approaches should carefully consider the level at which data are 
presented so that such concerns may be mitigated. 
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Both students and instructors also kept returning to the idea of time, which McPherson et al. (2016) also 
found in their recent study of Australian students' perspectives on data use. Students wanted help with 
scheduling and time management and saw the value of a system that could help them with 
organization. They thought it would be beneficial to use time-based data to optimize their study habits. 
Instructors similarly wanted information about how students were spending time outside of class. 
Although such time-based data are not readily collected, it represents a potential avenue for creative 
learning analytics research that potential users of such systems identified. If potential users do not view 
currently collected class-based data as helpful or exciting, this suggestion of focusing on time data could 
point learning analytics researchers in an empirically driven direction. 

Finally, one key difference between the groups of stakeholders is that students and instructors tended 
to emphasize different settings. Instructors focused on data related to their specific class or data that 
could help them understand students on a personal level, with the objective of tailoring their curricula 
to students' interests. Instructors ultimately linked each suggested data element to helping students 
find success in a specific course. Students, rather, kept taking the conversation away from the 
curriculum to broader life scenarios. They wanted help with time management or how to better 
visualize their big picture needs across all of their classes and activities. Perhaps this finding is not 
surprising, but learning analytics researchers and developers should keep in mind that the definition of 
"success" likely varies across stakeholders. Faculty members might view student success within their 
specific courses, but students might have a broader view in that they are thinking about success in their 
programs or majors. Designing learning analytics systems for specific stakeholders with the appropriate 
"success" definition in mind is essential for drawing users to the system initially. If students, for 
example, see a value-add in a time management system, perhaps a few mouse clicks into the system can 
lead them to the course-specific learning analytics views that researchers and developers tend to focus 
on first. Following such an approach to learning analytics research that interrogates student and 
instructor users throughout the design process can help uncover such perspectives so that uptake of the 
ultimate product may be more widespread and sustainable. 
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